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Abstract 

This article provides an overview of the current state of repositories of open educational resources 
(ROER) in higher education at international level. It analyses a series of educational indicators to 
determine whether ROER can meet the specific needs of the education context, and to clarify 
understanding of the reuse of open educational resources (OER) provided by ROER. The aim of the 
study is to assess ROER by combining these two perspectives, and to form a basis for discussion among 
the universities that are responsible for these repositories. 

The method was based on content analysis and consisted of two phases: an exploration of international 
sources, and an analysis of 110 ROER using the proposed set of indicators. The results focus on data 
from the analysis of ROER websites and some models of good practices. They are presented according 
to three core dimensions for evaluating ROER: general factors to establish types of ROER, a focus on 
drivers for OER reuse, and a focus on educational aspects. It was found that most of the ROER that 
included one or more of the proposed reuse indicators were created exclusively for educational 
resources. Educational aspects are not yet firmly embedded into ROER. The few repositories that seem 
to have successfully included them are those that provide other educational metadata and use 
educational standards. 
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Introduction 

In the past decade, the Open Educational Resources (OER) movement has gained ground and expanded 
exponentially. Previously, there were only localized OER initiatives; now OER are recognized and 
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supported by key international institutions, governments and funders worldwide. Although OER have 
become more significant worldwide and their availability and use have expanded, the movement has 
still not achieved its full potential and entered the mainstream education system (Ochoa & Duval, 2009; 
Dickev & Dicheva, 2012; European Commission, 2013a). 

In February 2015, the collaborative report “OER Strategy Development” (Allen, Browne, Forward, 
Green, & Tarkowski, 2015) provided a concise analysis of the current state of the OER movement and 
became a roadmap to identify specific strategies to achieve the “real adoption" of OER (understood as 
taking advantage of the rights and permissions granted by OER for use and reuse). 

According to the report, current platforms that enable the management, discovery, use, and reuse of 
open content are inadequate and not very well-known; resources are found in a wide range of formats, 
and repositories are varied and generally do not include common search terms or metadata. For this 
reason, a study on the extent to which existing OER repositories (ROER) promote the discovery, use, 
and reuse of resources may lead to improvements in the adoption and impact of OER. 

The OER movement in Europe has led many universities to rethink their institutional policies, and 
consider how to innovate in higher education teaching and learning practices. In its Opening up 
Education initiative, the European Commission (2013a) observed that “stimulating supply and demand 
for high-quality European OERs is essential for modernizing education” and that “OER must become 
more visible and accessible to all citizens.” One of the key transformative actions proposed is to improve 
the visibility of quality OER produced in the EU by 2020, through the development of open digital 
ROER using investment funds. 

Evidence (European Commission, 2013b) shows that greater exploitation of OER would increase access 
to education, and the development of quality educational standards would have a positive effect. 
Consequently, ROER should include educational aspects and metadata so that the most relevant 
resources are classified and can be retrieved by the main users: teachers and students. 

This study takes into account a series of educational indicators that determine whether ROER can meet 
the specific needs of the education context. It also defines the scope of ROER and proposes a series of 
indicators to analyze them, and particularly to clarify understanding about the reuse of OER provided 
by ROER. Consequently, this study could help to determine what can be done to improve ROER and 
may become a basis for discussion among universities and institutions responsible for these 
repositories. This is deemed essential, since ROER can make a substantial contribution towards 
achieving the goals of the Europe 2020 strategy: quality and relevance in higher education (European 
Commission, 2014). 


Background 

Global knowledge and research in OER is increasing. This topic has aroused the interest of researchers, 
as OER could make a strong contribution to improving education and fostering openness. 
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One of the main themes of OER research is OER and technology, and repositories in particular 
(Zancanaro, Todesco, & Ramos, 2015). A pioneering, complete overview of ROER that examined their 
main features was conducted by Tzikopoulos, Manouselis, & Vuorikari (2007). Several other studies 
focused on guidelines for developing ROER (Organisation for Economic Co-Operation and 
Development [OECD], 2007; UNESCO, 2011), their structure (McGreal, 2011), operational aspects and 
a quantitative analysis of repositories (Ochoa & Duval, 2009), additional services that they might 
provide (Zervas, Alifragkis, & Sampson, 2014; de los Arcos, Farrow, Perryman, Pitt, & Weller, 2014) 
and a holistic approach to quality assurance (Atenas & Havemann, 2013; Clements, Palowski, & 
Manouselis, 2015). Similarly, a recent study focused on the institutional promotion of OER through 
repositories (Castano, Punie, Inamorato, Mitic, & Morais, 2016) and details of ROER have been 
gathered in international directories, such as the OER World Map (D’Antoni, 2013), OER Repositories 
World Map (Atenas & Havemann, 2014) and OER Atlas (Neumann & MuuB-Merholz, 2016). 

However, none of these studies specifically refers to the reuse and educational aspects of ROER: these 
two unresolved issues merit particular attention in our research. 

Wiley (2000) considered that any digital resource could be reused to support learning, and subsequent 
studies have stressed the importance of reuse. However, they have also recognized the great difficulty 
in finding evidence of reuse (Orr, Rimini, & Van Damme, 2015), the still insufficient analyses (de los 
Arcos et al., 2014), and the “problem of remix” (Wiley, Bliss, & McEwen, 2014). The most common 
discussion framework has been focused on user surveys (McKerlich, Ives, & McGreal, 2013; Camilleri, 
Ehlers, & Pawlowski, 2014) and tracking OER usage (Pegler, 2011). A couple of studies specifically 
address the connection between reuse and ROER as a key theme that represents one type of activity 
within a repository (Atenas & Havemann, 2013), or link reuse to the creation of personal spaces (Cohen, 
Reisman, & Sperling, 2015). Nevertheless, ROER’s current “predisposition” to reuse has not been 
considered. For this reason, the present research examines opportunities provided by ROER that 
facilitate reuse of OER. 

Regarding educational issues, studies have mainly focused on benefits, challenges, and limitations 
associated with OER (Wiley et al., 2014), their creation and distribution (open textbooks, repositories, 
OCW courses and MOOCs), and their important role in open education (Lane & McAndrew, 2010; 
UNESCO, 2011; OECD, 2015). There are many studies on metadata evaluations of ROER (Bueno-de-la- 
Fuente, Hernandez-Perez, Rodriguez-Mateos, Mendez-Rodriguez, & Martin-Galan, 2009), but these 
adopt a formal and technological perspective rather than analyze the educational relevance of ROER 
(Rodriguez, Dodero & Sanchez-Alonso, 2011; Atenas & Havemann, 2013). Therefore, the current study 
aims to explore how ROER support teaching and learning, based on educational requirements. 

We are aware of the connection between reuse and educational aspects, and understand that reuse can 
be more difficult because the educational design is rarely visible (Wiley et al., 2014). Therefore, this is 
the first review that integrates both perspectives in ROER. It also provides an overview of the current 
state of ROER in higher education. 

As some aspects of ROER are underdeveloped, a distinctive, value-added framework is becoming 
indispensable. Since reuse and educational aspects in ROER still pose certain challenges, we propose 
the following research questions: 
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1. What are ROER and what are their main features? 

2. What features can be found in ROER that promote reuse and educational aspects? 

3. How developed are ROER with respect to reuse and educational aspects? 


Method 

The method used in this study was based on content analysis (Krippendorff, 1980) and consisted of two 
phases: 

1. Exploration of international and specialized sources about ROER, and selection of the ROER 
population to be analyzed. 


2. Analysis of the ROER population using a proposed set of indicators, divided into three main 
categories that were directly related to the research questions. 


Selecting a Population 

In order to select ROER for analysis, the following sources were used: the Registry of Open Access 
Repository (ROAR), the Directory of Open Access Repositories (Open DOAR), the portal Open 
Education Europe , a report on the State of the Art by the Educational Repositories Network (EdReNe), 
and other literature on the subject. 

After retrieving a total of 1,186 repositories of “learning objects” or “educational resources,” a 
population of 110 ROER was selected using the following criteria (Figure 1): 

A. A higher education (HE) level of education, or HE combined with other levels. 

B. Updated from 2011 onwards, and currently functioning. 

C. Repositories with specific OER collections. 

D. Repositories containing at least 50 OER. 


87 








Repositories of Open Educational Resources: An Assessment of Reuse and Educational Aspects 
Santos-Hermosa, Ferran-Ferrer, and Abadal 


•ROAR 


SOURCES: 

International Repositories 


•Open DOAR 

•Open Education Europe 

•EdReNe 


SELECTION CRITERIA 


•A) Higher Education level 
• B) Updated content 
•C)Specific OER collection 
•D) Minimum number of OER 


•110 Repositories with an OER 
collection in Higher Education, 
POPULATION provided that there are >50 

resources which are from 2011- 
2016 



Figure l. Selection of the ROER population (Santos-Hermosa, Ferran-Ferrer, & Abadal, 2015). 

Core Dimensions and Indicators 

Repositories tend to be evaluated using institutional factors that are based on institutional needs and 
strategies (Westell, 2006; Serrano, Melero & Abadal, 2014). However, considering that ROER’s 
particular requirements differ from those of traditional digital repositories (Cervone, 2011), other 
aspects could be taken into account. 

The current study proposes three core dimensions for evaluating ROER: general/descriptive factors to 
establish types of ROER, a focus on drivers for OER reuse, and a focus on educational aspects. These 
categories are broken down according to specific indicators. 
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Category 1: General indicators: Types and technology. 


Table l 

ROER Types and Technology 


CRITERIA 

INDICATORS 

DESCRIPTION AND VALUES 

REASONING & REFERENCES 

OER stored wholly or 
partially 

Exclusive ROER 

Only contains learning resources, created 
especially for OER 

In-house indicator that might identify 
particular patterns in ROER. 

Hybrid ROER 

Contains research and learning resources 

Discipline 

Multidisciplinary 

Stores OER from all disciplines 

Indicator for discovering the issues 
adressed in ROER: broader vs. subject 
domain (Sampson, Zervas & Sotiriou, 
2013) 

Thematic 

Stores OER from one or specific disciplines 

Person responsible or 

creator 

Institutional 

Created by universities, foundations or 

institutes 

Indicators about authorship and origin 
have been considered in studies 
evaluating ROER (Tzikopoulos, 

Manouselis & Vuorikari, 2007; Atenas & 
Havemann, 2013) 

National 

Funded by governments, national 

organizations or consortia 

Geographical origin 

Continent 

Africa, America (USA & South America), Asia, 
Australia & Europe 

Country 

Name of the country 

Bueno-de-la-Fuente et al., 2009 

Software/platform 

Name of software 

Identifies the software for repositories 
(Dspace, Eprints, Fedora, etc) or other kinds 
of platforms (Moodle, html5, CMS, Drupal) 

Although an institutional approach is not 
entirely relevant, general indicators 
related with software and metadata 
have been considered (Bueno-de-la- 
Fuente et al., 2009; UNESCO, 2011; 
Pegler, 2011; Barrueco etal., 2014). 

Metadata standard 

Name of the 

metadata 

standard 

Identifies the metadata standard system 
used for indexing OER: Dublin Core, SCORM, 
LOM, free tags, etc. 


Category 2: Indicators of reuse. One of the benefits of ROER is that teachers have the 
opportunity to “reuse, modify, and adapt resources” (Hylen, Damme, Mulder, & D’Antoni, 2012), and a 
secure and reusable platform is an important design factor that contributes to reuse (Wills & Pegler, 
2016). Hence, ROER that implement drivers contributing to reuse would become “beneficial reusable 
platforms” for users. Table 2 shows a proposal of facilitating factors, and gives reasons for this chosen 
focus. 
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Category 3: Educational indicators. Since it has been emphasized that the pedagogical 
value of educational resources is the most important factor (McGreal, 2004), Table 3 proposes a series 
of indicators that are designed to describe ROER from an educational perspective. 


Table 3 

Educational Indicators 


INDICATORS 

DESCRIPTION 

VALUES 

REASONING & REFERENCES 

OER type 

Information about the types of OER, 
identified through specific metadata, 
filters or limiters that allow searches 
for types of resources. 

Yes/No 


Educational 

metadata 

Specific metadata for educational 
aspects are provided. 

Educational level/grade 

Educational metadata can be used to 

assess the educational relevance of 
ROER (Rodriguez et o/., 2011; Atenas 
& Havemann, 2011). However, 
because of the imprecision and poor 
quality of metadata records, the 
content is not fully validated 
(Palavitsinis, Manouselis & Sanchez- 
Alonso 2014). 

Educational context 

Intended/suggested use 

Learning time 

Intended audience 

Pedagogy 

Syllabus/curriculum 

No specific metadata for education 

Learning goals 

Specific metadata for learning goals 
are provided. 

Yes/No 

Supported by the users 7 need in ROER 
for "detailed information related to 
each OER and objectives for using 
them must be given" (Yalcinalp & 
Emiroglu, 2012). 


Data Gathering and Analysis 

Content analysis was carried out for each of the 110 repository websites, using Tables 1 to 3 as 
codification frameworks. Tools for retrieving information from each website were used, such as 
searching (filters and metadata) and browsing (documentation). Data were collected between 2015 and 
2016. 

Content analysis limitations were broken URLs, duplicate data and the omission of pertinent data. 
Another limitation of this study was that some repositories are available in other languages, and have 
not been translated into English. 


Results and Discussion 

This section presents and discusses data from the analysis of ROER websites. In addition, some relevant 
ROER are highlighted as models of good practices. 

ROER Types and Technology 
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Main features. Table 4 indicates that there are more OER repositories that only contain 
learning resources (54.5%) than hybrid repositories (45.5%) containing research and learning 
resources. 

Table 4 

OER Stored Wholly or Partially 


Typology 

Total 

% 

Hybrid repositories 

50 

45.5% 

Exclusively OER repositories 

60 

54.5% 


Most ROER (Tables 5 and 6) were multidisciplinary (73.6%). The second most common discipline in 
ROER was Science, Technology, Engineering, and Maths (STEM, 12.7%). Most ROER were dedicated 
to higher education (85.5%); a lower number covered different levels of education (14.5%). Although a 
multidisciplinary approach also seemed more common in a previous analysis (OECD, 2007), other data 
suggest that single-subject repositories would be more suitable for teachers, since it is harder for 
teachers to find and use relevant OER for their subjects in broader repositories (Sampson, Zervas, & 
Sotiriou, 2013). 

Table 5 

Disciplines 


Disciplines (HEA Academy) 

Total ROER 

% 

Multidisciplinary 

81 

73.6% 

STEM - Science, Technology, 
Engineering and Mathematics 

14 

12.7% 

Social Science 

10 

9% 

Arts & Humanities 

3 

2.7% 

Health & Social Care 

2 

1.8% 


Table 6 

Educational Level 


Typology 

Total 

% 

Higher education 

94 

85.5% 

All levels of education 

16 

14.5% 
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Authorship. It was found that the prevailing ROER were institutional (76.4% created by 
universities, foundations or institutes) and the rest were national (23.6% funded by governments, 
national organizations or consortia). This pattern is similar to that found in another analysis (Atenas & 
Havemann, 2013), in which institutional ROER were also predominant, but to a lesser extent (50% of 
the total, while the rest was broken down into different types). 

Geographical origin. As stated in the literature (Tzikopoulos et al., 2007; Atenas & 
Havemann, 2013), most ROER were created in Europe (72.7%) and North America (16.3%) (see Table 
7). Others were scattered between South America (6.3%), Asia (3.6%) and Africa (0.9%). Twenty-seven 
countries were represented in the total ROER population (Table 8). The countries that created the 
highest numbers of ROER were the UK (18.1%), Spain (15.4%), the USA (13.6%) and France (10%). 

The success of ROER in the UK might be due to the considerable funding provided for OER projects by 
the Joint Information Systems Committee (JISC). This pattern varies in other studies (Bueno-de-la- 
Fuente et al., 2009), in which the USA was found to be the most common location (28%), followed by 
Germany and Italy. However, our results might be influenced by the fact that they only focus on higher 
education ROER. The countries that were found to be most involved in ROER may coincide with the 
countries in which most publications about OER have been identified (Zancanaro et al., 2015). 

Table 7 

Geographical Origin of ROER (Continents) 


Continents 

Total 

% 

Europe 

80 

72.7% 

Asia 

4 

3.6% 

North America 

18 

16.3% 

Africa 

1 

0.9% 

South America 

7 

6.3% 
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Table 8 

Geographical Origin ofROER (Countries) 


Countries 

Total 

ROER 

% 

UK 

20 

18.2 

Spain 

17 

15.5 

US 

15 

13.6 

France 

11 

10 

Germany 

9 

8.2 

Colombia 

4 

3.6 

Poland 

Brazil 

3 

2.7 

Greece 

Ukraine 

Mexico 

2 

1.8 

Portugal 

Belarus 

Indonesia 

Canada 

1 

0.9 

China 

Estonia 

Finland 

Kenya 

Belgium 

India 

Ireland 

Italy 

Slovenia 

Russian Federation 

Sweden 

Switzerland 

27 

110 

100% 


Software platform. Data on the technology that supports ROERs platforms (Table 9) 
suggests that DSpace is the most popular software (34.5%), followed by a range of other software 
(19.1%), some content management systems (CMS) and learning management systems (LMS) (11.8%), 
and Eprints (7.3%). Other software that is used to a lesser extent is dLibra, Equella, Fedora, and 
MyCore. However, in a significant number of ROERs (14.5%), the platform was not identified, as it was 
not mentioned in their policies or anywhere else. This result coincides with that of OpenDOAR and 
Bueno-de-la-Fuente (2009), although recent studies (Amiel & Soares, 2016) have indicated a tendency 
to adopt CMS in the ROER scene. 
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Table 9 

Software Platform 


Repository software 

Total 

% 

Dspace 

38 

34.5% 

Others (NClor, Hydra, OMEKA, 

Run C0C0, Miless, DiDoRe, etc.) 

21 

19.1% 

Not identified 

16 

14.5% 

Web/CMS/LMS 

13 

11.8% 

Eprints 

8 

7.3% 

dLibra 

4 

3.6% 

Fedora 

4 

3.6% 

EQUELLA 

3 

2.7% 

MyCoRe 

2 

1.8% 


Metadata standards. The indexing of resources using metadata facilitates their location, 
dissemination and harvesting from other online platforms. In this analysis, the most widely used 
metadata standard in ROER was Dublin Core (DC, 50.9%). It was not specified whether simple or 
qualified DC was used, but both types may be involved, since additional qualifiers that further refine 
the meaning of OER have been found. The predominance of the DC element set might be due to the fact 
that DSpace uses it as its base metadata schema and DSpace was the most popular software used in 
ROER, as shown above. 

However, the DC schema cannot efficiently characterize educational material or meet the needs of 
ROER (Koutsomitropoulos, Alexopoulos, Solomou, & Papatheodorou, 2010). Accordingly, several 
educational metadata standards have appeared over time, to improve the description of educational 
resources by identifying their pedagogical properties. A widely adopted standard is IEEE LOM 
(Learning Object Metadata), which has an inherent extension capability (Al-Khalifa and Davis, 2006) 
and attributes that enable learning resources to be managed and evaluated (Nair & Jeevan, 2004). Other 
learning specifications are IMS, which is a development of LOM, and SCORM (Sharable Content Object 
Reference Model) for sequencing structured educational material in the form of reusable content. 
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Table 10 

Metadata Standards 


Metadata standard 

Total 

% 

Dublin Core 

56 

50.9 

Not identified 

19 

17.3 

LOM/SupLOMFR 

16 

14.5 

Free Tags 

10 

9.1 

Other 

4 

3.6 

METS/PLMET 

3 

2.7 

SCORM 

1 

0.9 

IMS QTI 

1 

0.9 

TOTAL 

110 



Twenty per cent of the ROER adopted specific educational medatada standards for describing the OER 
(table n). The most commonly used standards were LOM/SupLOMFR (14.5%), SCORM (0.91%), IMS 
(0.91%) and others (3.6%) (see Tables 10 and 11). Some other ROER provided a free tag system (9.1%). 
Lastly, 2.7% used a generic schema for describing digital objects, called METS/PLMET. 

Table 11 

Educational Metadata Standard 


Educational metadata 
standard 

Total 

LOM/SupLOMFR 

16 

SCORM 

1 

IMS QTI 

1 

Other educational standard 

4 

TOTAL 

22 (=20%) 


The pattern shown by these data contrasts with the findings of other studies, in which the importance 
of IEE LOM (29% of ROER) outstripped that of DC (22%) (Tzikopoulos et al., 2007) and in which less 
volume and fewer details of the metadata standard (37.7% included either DC or LOM) were identified 
(Atenas & Havemann, 2013). 

Reuse in ROER 

Intentionality. A total of 36.3% of ROER expressed an intention to reuse materials, in the 

form of 
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• comments in the mission or in the collection’s policies, which were found in ARIADNE, the 
University of Leicester’s OER Repository or Xpert, for example; 

• advice and instructions on the reuse of resources. Open Michigan has a “Use Open Content” 
section, with recommendations on how to adapt and license resources; and 

• messages encouraging users to remix resources. NCLOR proposes that teachers can become 
more productive and save valuable time if they contribute new versions of resources. OER 
Africa also encourages teachers to adapt materials to African contexts and learners. 

Table 12 

Intention to Reuse 


Intention to reuse 

Total 

% 

Yes 

40 

36.3% 

No 

70 

63.6% 

TOTAL 

110 



Versioning. Thirteen of the ROER used some technical implementations to record different 
versions of the same resource. For instance, Jorum allows "multiple versions of an object to be deposited 
by adding information on the version in the description.” Other ROER provided the following. 

• Search filters to retrieve available versions, including information on where a specific resource 
was used (RADAR), how to reuse resources (HUMBOX), and where to find resources to remix 
or make derivatives (OER Commons). 

• Specific metadata and sections for versioned resources, including “Life cycle” or “Details” 
(CURVE, Dashboard Equella and Banco de Objetos Virtuales de Aprendizaje), “Version 
History” (OpenStax CNX) and “Copy history based” (TEMOA). 

• The most developed ROER showed chronologically who, when, and which files had been 
changed (DuEPublico); offered assistance about how to reuse or create contents that are 
suitable for reuse (in TEMOA “you can mix and match other people’s items to build new ones 
and adapt them to your needs”) and recommended how to reuse and attribute educational 
content (OpenStax CNX). 

Of the subtotal of ROER that described reuse intentions (40 out of 110 repositories) (Table 13), only 
32.5% (13 out of 40 ROER) included versioning. Therefore, a more intentional rather than a technical 
solution to support reuse has been observed. This may be because maintenance and updating are 
problems that ROER are struggling with (McGreal, 2011). 
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Table 13 
Versioning 


Repositories with REUSE 

Total 

% 

Intentionally 

27 

67.5% 

Through versioning 

7 

17.5% 

Intentionally & through versioning 

6 

15% 

TOTAL 

40 



Licenses. Table 14 shows that only 5.5% of repositories had restricted access to resources 
(copyright with all rights reserved or login requirements). A total of 56.4% had an open license, 24.5 
had both open and restricted licenses and, finally, 13.6% did not specify the license. 

Amongst the open licenses, Creative Commons (CC) prevailed (43.6%), followed by specific licenses for 
educational use (10%), and other in-house licenses provided by the repositories (2.7%). As for ROER 
with a combination of both open and restricted access, 11 out of 27 were found to have CC. 

Table 14 

Licenses 


License access 

License type 

ROER 

% 

Open 

CC 

48 

43.6 

For educational use 

11 

10 

In-house license 

3 

2.7 

Subtotal Open 

62 

56.4 

Both open & 
restricted 

Not identified 

16 

14.5 

CC 

11 

10 

Subtotal Open & Restricted 

27 

24.5 

Restricted 

6 

5.5 

Unspecified 

15 

13.6 


110 



In some studies, 76% of ROER had copyright policies (Tzikopoulos et al., 2007) and 52.5% (Atenas & 
Havemann, 2013) or 22% (Amiel & Soares, 2016) had CC licenses. Our results were more positive, since 
they showed an increase in openness of ROER and in the use of CC. However, there were still very few 
CC licenses, considering that they are relatively easy to adopt and lead to greater access, use and reuse 
of resources. 

Our findings also revealed that CC are more commonly implemented in the USA and Europe, where 
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more ROER are based. The opposite situation was found by Atenas & Havemann (2013), where the 
incidence of CC was greater in areas with fewer ROER (in Oceania, the Middle East and Africa). 

The specific type of CC license provided by ROER can reveal the real opportunities they provide to reuse 
resources. Out of a total of 59 ROER with CC (Table 14), we observed (Table 15) that most allowed reuse 
(86.4%): 42.4% were Attribution-Noncommercial-Share Alike (BY-NC-SA), 27.1% used any of the six 
CC licenses, and 16.9% were under BY-NC. This last license allows others to non-commercially remix 
the original work, and, if it is also combined with SA (BY-NC-SA), new creations that build on the 
original work must be licensed under identical terms. The remaining 13.6% that did not allow any reuse 
corresponded to the most restrictive CC (BY-NC-ND), which only permits others to download and share 
resources if they credit the author, and does not allow the material to be changed in any way or used 
commercially. 

Table 15 

Creative Commons Licenses 



CC license 

Number 

% of ROER 
with CC (59) 

CC that allow reuse 

BY-NC-SA 

25 

42.4 

Any of the 6 CC licenses 

16 

27.1 

BY-NC 

10 

16.9 

Subtotal allowing reuse 

51 

86.4 

CC that do not allow 

reuse 

BY-NC-ND 

8 

13.6 

Subtotal not allowing reuse 

8 

13.6 


Total CC 

59 



The preponderance of the CC-BY-NC-SA license is in line with the findings of other studies (Venturini, 
2014; Amiel & Soares, 2016), but was clearly higher in the current research (42.4% versus 8% found by 
Amiel). 

Although the number of CC licenses that facilitated reuse was high (51 out of 59), it still only represented 
less than half of the total ROER (51 out of 110). 

Granularity. It was found (Table 16) that just 20.9% of ROER had resources with different 
levels of granularity. Other research has shown that granular resources are more valued and effective 
for users (Wharrad & Windle, 2010) and play an important role in reuse for teaching and learning 
(Yalcinalp & Emiroglu, 2012). 

Learning Space specifies that resources can be catalogued both as single entities (an entire manual) 
and constituent parts (chapters). Temoa also refers to resources as topics, activities or courses. ATE 
Central provides separate categories for ‘’’instructional unit” and “courses;” RUA provides specific 
metadata ( c.relation.ispartof) to indicate that modules are part of a subject or course, and RADAR links 
the different units related to a course. 
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Table 16 
Granularity 


Typology 

Total 

% 

Higher education 

94 

85.5% 

All levels of education 

16 

14.5% 


Granularity and the rest of the indicators of reuse (Table 17) were found to be interconnected. Thus, 
ROER with granularity also included social networks (78.3%), a quality system (73.9%) and some 
intention for reuse or versioning (60.9%). 

Table 17 

Indicators of Reuse in ROER with Granularity 


ROER with granularity (23) and: 

Number 

% 

Social networks 

18 

78.3 

Quality system 

17 

73.9 

Reuse intentional/versioning 

14 

60.9 

Comments or ratings 

13 

56.5 

CC for reuse 

11 

47.8 

Open formats 

2 

8.7 


Open formats. A total of 154% of the ROER included open formats (Table 18). Amongst 
these, only 9.1% implemented open formats as metadata; the other 6.3% included open formats, but 
they did not appear in the results. 

Table 18 

Open Formats 


Open formats 

Total 

% 

Not included in metadata 

93 

84.5% 

Included in metadata 

10 

9.1% 

Open formats amongst results 

7 

6.3% 


Ten of the 17 ROER with open formats also supported reuse (intentionally or through versioning), and 
14 had open licenses (most of them were CC; some had licenses for educational use). These results are 
fairly consistent if we consider that open formats should facilitate reuse. 
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Some good practices regarding open formats in ROER were the inclusion of recommendations about 
their adoption, which were found in some guidelines: “attention should be given to the adoption of open 
standards. We would generally recommend resource authors to use open source solutions in preference 
to proprietary ones” C ReStore ); “open formats ensure the preservation of resources” (Eprints UCM) and 
“whenever is possible, the open and easily re-usable formats should be used” (Oregon’s repository). 
Finally, MERLOT is committed to creating materials with Content Builder and Pachyderm open 
software. 

Some preferred formats that were detected are: .odt or .txt for text and .flac for audio (Oregon’s 
repository), epub for ebooks (Open Learn), html/xml ( DLynx, UVED, OpenStax CNX and UOH ) and 
Css for web (Jorum). 

Quality. A total of 43.6% of the ROER specifically mentioned quality, from a top-down 
perspective and according to three types of criteria (Table 19). This result contrasts with that of other 
studies (Castano et al., 2016), in which quality assurance was even less present (11%). 

Table 19 

Top-Down Quality Criteria 


Top-down quality indicators 

Detail 

Total 

% 

WITH CRITERIA 

Internal criteria 

Provided by the institution 

22 

20 

Author prestige 

Provided by the authors themselves 

15 

13.6 

External criteria 

Peer review 

8 

7.3 

Group of experts 

2 

1.8 

Award or Quality labels 

2 

1.8 

Subtotal 

11 

10 

Subtotal of top-down criteria 

48 

43.6 

WITHOUT CRITERIA 

No criteria 


4 

3.6 

Unspecified 


58 

52.7 

Subtotal of no criteria 

62 

56.4 


TOTAL 


110 



According to our data, the most common approach when there is a form of quality assurance is internal 
assessment (20%), followed by the prestige of authors (13.6%) and external reviews (10%). This pattern 
is aligned with UNESCO (2011), which states that institutions are responsible for assuring the quality 
of OER used in teaching and learning environments, and should invest in improving quality before 
making materials available in ROER. The peer review, which is used extensively in scholarly publishing, 
has also become a quality assurance measure in ROER (Clements et al., 2015). 

Institutions often ensure quality through the OER submission process (publishing rules) rather than 
the content. However, quality parameters exist in some cases, for example in the JISC Enriching Digital 
Resources Programme (in the First World War Poetry Digital Archive). Other ROER use guidelines 
that guarantee relevant OER: those with “learning objectives, instructional strategies and alignment 
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with educational standards” ( Exploratorium's Howtosmile), and with “an eminent educational or 
research purposes in teaching” ( Open SNH ). In other cases, a review checklist is provided before 
resources are uploaded (Humbox). 

The main reasons given by ROER that provide quality through recognition of the author are that the 
authenticity of the content is the sole responsibility of the creator, and authors have been full-time 
lecturers for other organizations and are experts in their fields. With these arguments, ROER exempt 
themselves from any responsibility for content, as authors must meet certain conditions before 
submitting OER (Temoa) or must participate in all the development stages ( Chem Collective). 

Some ROER based on peer review systems rely on specialist groups ( Learning Exchange ) or editorial 
boards (Merlot) that decide which resources are suitable for their communities. Others make public 
their review criteria, such as “content quality, scope, beneficiaries, usability and potential ongoing value 
of the site” C Restore ) and “high scientific accuracy, good pedagogical effectiveness, ease of use, clarity 
and completeness of documentation, motivating for learners, show robustness, and illustrate 
significance of content” (DLESE). 

The few ROERs (3.6%) that do not explicitly assume liability for their resources (Table 19) are those 
stored in external containers (such as WLP), or not subject to quality control ( DuEPublico ). 

A bottom-up quality approach based on user-generated evaluation (by rating and comments) will be 
analyzed in the next section. 

Community of users. A total of 62.7% of ROER (69 out of 110) provided some kind of 
community engagement (in the remaining cases, this was an unknown factor). The most common 
interaction allowed was recommending OER by sharing them through social networks, electronic mail 
or RSS. This was found in approximately half of the ROER (54 out of 110). Conversely, less than one- 
quarter of ROER allowed comments (19 out of 110) or ratings (12 out of 110). Thus, in terms of ensuring 
a bottom-up quality system based on user action, ROER took less account of evaluation tools, such as 
rating, than of social media services. 

Table 20 

Community of Users (Bottom-Up Quality Criteria) 


Bottom-up quality criteria 

Type of Community 
engagement 

Total 

% 

Social interaction 

Sharing 

54 

49.1 

Comments 

10 

9.1 

Comments & ratings 

5 

4.5 

Total social interaction 

89 

62.7 

Unspecified 

Unspecified 

41 

37.3 


TOTAL 

no 



These findings are similar to some previous results (Atenas & Haveman, 2013), in which 51.2% of ROER 
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allowed sharing, and 32.5% allowed ratings. However, it is difficult to discern what these numerical 
scores mean: are they positive enough to claim real community engagement or do they demonstrate the 
existence of review systems for users? Some authors (Zervas et al., 2014) found that social 
functionalities were not strong enough to assure quality (Ochoa & Duval, 2009). Clement (2016) added 
that they were insufficient if unsupported by a users’ community. At the moment, these results show 
the extent to which community systems are available in ROER. 

In a more qualitative analysis, some examples of ROER point to the existence of community spaces and 
behaviours. These include: discussion areas and special interest groups (in Promethean Plane t); groups 
of contributors that use and comment on OER (in Humboldt Digital Scholar and LORO); communities 
of interest and disciplines with their own YouTube channel and blogs for registered users (in MERLOT); 
and international communities involved in creating and adapting OER and participating in a blog and 
wiki (in Open Michigan). Other ROER have recognition systems to evaluate and receive recognition 
from the community. For example, some assess members’ level of contribution (TEMOA) or provide 
"top members" ratings (Promethean Plane t). 

Finally, a comparison of all the proposed indicators of reuse (Table 21) shows that while they do not 
have enough presence in ROER (all figures were below 50%), except licensing and social networks (close 
to or over 50%), most have a greater incidence in OER-exclusive repositories. In these, the features that 
were found most frequently were again licensing and social networks (with an increment of 20.9 points), 
as well as quality, intention and versioning. Note that although the figure for granularity is still low 
(38.3%), it is much higher than in the total ROER (17.4 points higher). 

Table 21 

Indicators of Reuse 


INDICATORS OF REUSE 

Total ROER 

Total OER-exclusive 
repositories 

% Difference (OER- 
exclusive vs. total ROER) 



Number 

% 

Number 

% 


Total intention 
& versioning 


40 

36.4 

32 

53.3 

17 

Licensing 

Open 

licenses 

62 

56.4 

37 

61.7 

5.3 


CC licenses 

59 

53.6 

38 

63.3 

9.7 

Granularity 


23 

20.9 

23 

38.3 

17.4 

Open formats 


10 

9.1 

6 

10 

0.9 

Quality 


48 

43.6 

33 

55 

11.4 

Community of 

Social 

networks 

54 

49.1 

42 

70 

20.9 

users 

Comments 
or ratings 

31 

28.2 

24 

40 

11.8 


Educational Aspects 

OER types. The ROER (85.4%) provided information about the type of stored resources (Table 
22). This information was usually found in the metadata “dc.format” or “dc.format.mimetype,” other 
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related metadata, or the search type drop-down list. 

Table 22 

Metadata on Type of Resources 


Type of resource 

Total 

% 

Yes 

94 

85.4% 

No 

16 

14.5% 


Table 23 shows the wide range of resources that were identified in the ROER. 
Table 23 
Types ofOERS 


Media Type 

Text 

Textbooks, book chapters, exercises, courses, modules, lessons, 
weblinks, webgraphies, blogs and wikis. 

Image 

illustrations, graphics, maps and photographs 

Audio-visual 

Videos, interactive multimedia and podcast. 

Interactive material 

Animations, games, self-evaluation, digital media and experiments. 

Software 

Simulators, demos or other tools and desktop applications. 

Educational Type 

Guides 

Guides, syllabus, glossaries, indexes and reference lists. 

Course material 

Course lectures, case studies, presentation, notes, datasets, 
conferences and slides. 

Assessment material 

activities, essays, assignments, quizzes, tests, exams, thesis, and 
final degree assessments 

Methods 

Teaching presentations, slides, etc. 


The content policies of the ROER tended to state what kind or resources could be included in the 
collections and who could include material. For instance, RODERIC asserted that “by teaching 
documents we mean any original material that may be of interest for use in the classroom” and 
LEEDSMET added that resources “may only be deposited by accredited members of the institution or 
their delegated agents”. In addition, Open Access policies establish that doctoral theses, final degree 
projects and dissertations must be deposited (PADEMOS and RIUMA ) and, in some cases educational 
resources must also be deposited ( UCREA ). 

Some ROER provided resources specifically about teaching and innovation in teaching as disciplines 
C DUGI Docs ) or offered specific collections on Curricula and Pedagogy (Belarusian National). Finally, 
PADEMOS provided a collection of “Material accessible to people with disabilities.” 

Educational metadata. A total of 42.7% of the ROER included specific metadata related to 
education (Table 24). The most popular educational metadata were “educational level or grade” (14.5%) 
and “intended or suggested use” (10.9%). Others were related to the intended audience, pedagogy, 
knowledge area, learning goals, learning time and related syllabus. Finally, in some cases in which there 
were no specific metadata for educational aspects, other information about targeted users or 
educational intention was located in fields designed for descriptions or comments. 
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Table 24 

Types of Educational Metadata 


Type of educational metadata 

Total 

% 

Educational level/grade 

16 

14.5 

Intended/suggested use 

12 

10.9 

Intended audience 

6 

5.4 

Pedagogy 

3 

2.7 

Education subdiscipline/knowledge area 

3 

2.7 

Learning goal 

3 

2.7 

Learning time 

2 

1.8 

Syllabus 

2 

0.8 

Total specific for education 

47 

42.7 

Nonspecific for education 

63 

57.3 


Regarding the type of standards used by ROER with and without educational metadata (Table 25), most 
of the ROER that used DC did not include specific educational metadata. In contrast, the vast majority 
of ROER that used educational metadata standards based on LOM or SCORM (19 out of 22) also 
provided metadata on specific educational aspects. This pattern could be seen as a logical consequence 
of the inherent characteristics of each metadata standard. Therefore, a greater presence of educational 
standards is expected in ROER that provide educational data, since these standards have been created 
with this purpose. 

Table 25 

Educational Standards and Metadata in ROER 


ROER 

DC 

Educational Standard 
{LOM, SCORM & IMS) 

Free 

tags 

METS 

Not 

identified 

Total 

% 

With educational metadata 

18 

18 

5 

3 

3 

47 

42.7 

Without educational metadata 

38 

4 

5 


16 

63 

57.3 

Total 

56 

22 

10 

3 

19 

110 



The results also showed another striking case: ROER with educational metadata using specific 
educational standards were equal in number to those using DC (18 in each case). This parity might be 
due to the possibility of mixing different metadata standards in the same repository, which is known as 
crosswalks metadata. This creates compatibility between descriptions, builds rich contexts for 
learning, and is useful for unnamed information that needs to be integrated (Godby, Young, & Childress, 
2004). For instance, LOM elements can be incorporated in the default DSpace’s qualified DC schema 
by direct mapping or by creating a further implementation (Koutsomitropoulos et al., 2010; 
Alexopoulos, Solomou, Koutsomitropoulos, & Papatheodorou, 2010; Skourlas et al., 2016). Therefore, DC 
metadata can be extended and optimized for education by incorporating educational descriptors. Some 
examples of metadata co-existing on the same platform can be seen in Jorum, which combines DC 
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(dc.audience), LOM (lom.educational.context) and JMD packages (jmd.oer) in the same register, and 
ARES UNASUS , which has its own metadata system based on LOM (f.i, unasus.typicalLearningTime). 

Learning goals. Just 29.1% of ROER included learning goals (Table 26). They were most 
commonly located in a description field (40.6%), but they were also found in specific descriptors for 
“Learning objectives” (37.5%) and “Instruction method/pedagogy” (9.3%) or included as part of the 
“Intended use” (3.1%) (see Table 27). 

Table 26 

Learning Goals 


Learning goals 

Total 

% 

No 

78 

71.9 

Yes 

32 

29.1 


Table 27 

Location of Learning Goals 


Location of the 
learning goal 

(9 Total | 

% 

In the Description 

13 

40.6 

Learning objectives/goals 

12 

37.5 

Intended/Suggested use 

3 

9.3 

Instruction/ 

Methodology/Pedagogy 

3 

9.3 

Other metadata 

1 

3.1 

TOTAL 

32 



Most of the ROER that excluded metadata on learning goals (55 out of 78) did not contain any other 
specific metadata on education. In contrast, ROER with learning goals as a descriptor also included 
other educational metadata (in 10 out 12) and used educational metadata standards (in 8 out of 12). 

Finally, by crossing all the proposed educational indicators (Table 28), educational aspects were found 
to have a higher incidence (with notable increases in educational metadata, which were present in over 
60%, and in learning goals in OER-exclusive repositories and a lower presence (all below 50%, except 
the type of OER) with respect to the total ROER. 


Table 28 

Indicators of Educational Aspects 
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INDICATORS OF 

EDUCATION ASPECTS 

Total ROER 
(110) 

Number % 

Total OER-exclusive 
repositories (60) 

Number % 

% Difference 
(OER-exclusive vs. 
Total ROER) 

Type of OER 

94 

85.5 

50 

83.3 

-2.1 

Educational metadata 

47 

42.7 

37 

61.7 

18.9 

Learning goals 

32 

29.1 

26 

43.3 

14.2 

Education standards 

22 

20 

18 

30 

10 


Level of Development of ROER 

Using the incidence of indicators of reuse and educational issues in ROER, we could identify the level 
of development of these repositories and establish four main types (Table 29). 

A. Top ROER: those that include 9 or 10 of the indicators (4 ROER). Only a few ROER intensively 
promoted reuse and educational aspects: Merlot (all 10 indicators) and OER Commons, UNIT 
and TEMOA (all three covered 9 of the 10 indicators). 

B. High ROER: those that include 7 or 8 of the total indicators (9 ROER). These repositories had 
a substantial presence in social networks, were OER type, and included educational metadata, 
open licenses and quality assurance. None of these repositories had open formats. The ROER 
in this category are: Banco de Objetos de Aprendizaje Colombia Aprende, BOB, NDLR, 
Prometheat, OpenLearn, ATE Central, CURVE Open, Open SHNH and Banco Internacional de 
Objetos Educacionais. 

C. Medium ROER: those that include 5 or 6 of the total indicators (31 ROER). The pattern is 
similar to that of High ROER, but the figures are considerably lower for some indicators, such 
as educational metadata. 

D. Low ROER: those that include 1 to 4 of the total indicators (66 ROER). In general, the figures 
for all the indicators were much lower, and in particular they lacked (or had more deficiencies 
in) open formats, granularity, learning goals and educational standards. 

Forty-four ROER (40% of the total) were placed in the top, high and medium categories, with different 
grades of promotion of reuse and educational features. However, the remaining 66 ROER, which make 
up the bulk of the population (60%), do not have enough drivers to facilitate reuse and educational 
issues. 
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Table 29 

Categorization of ROER According to the Incidence of Reuse and Educational Indicators 


TOTAL OF ROER 
(BY LEVEL) 

Intentionality 
& versioning 

Open license 
or CC 

Granularity 

Open 

formats 

Quality 

Social 

interaction 

OER 

types 

Educational 

standard 

Educational 

metadata 

Learning 

goals 

TOP 

4 

4 

4 

4 

2 

4 

4 

4 

3 

4 

4 

100% 

100% 

100% 

50% 

100% 

100% 

100% 

75% 

100% 

100% 

HIGH 

9 

6 

8 

5 

0 

8 

9 

9 

3 

9 

7 

66.7% 

88.9% 

55.6% 

0 

88.9% 

100% 

100% 

33.3% 

100% 

77.8% 

MEDIUM 

31 

18 

20 

10 

5 

19 

20 

22 

8 

19 

13 

58.1% 

64.5% 

32.3% 

16.1% 

61.3% 

64.5% 

71% 

25.8% 

61.3% 

41.9% 

LOW 

66 

12 

37 

4 

2 

16 

34 

55 

5 

11 

5 

18.2% 

56.1% 

6.1% 

3 

24.2% 

51.5% 

83.3% 

7.6% 

16.7% 

7.6% 


An initial table was used to classify ROER according to the incidence of reuse and educational 
indicators. It contained the population of 110 ROER in the X-axis, and the 10 assessment indicators in 
the Y-axis: 6 on reuse (intentionality & versioning, open licenses or CC, granularity, open formats, 
quality and social networks) and 4 on educational aspects (OER type, educational standard, educational 
metadata and learning goals). A second table, presented above, was drawn up to cluster similar ROER 
(those which included more or fewer indicators) and to identify the most popular indicators and the 
correlations between them. This is a basic, exploratory categorization that will be analyzed in greater 
depth in future research. 


Conclusions 

Based on the analysis, it has been found that most ROER on higher education are institutional (mainly 
created by universities and government bodies) and exclusively designed for educational resources, 
rather than hybrids that also contain research content. Considering that OER production remains 
essentially linked to public funding, international projects and institutions (Castano et al., 2016), it 
makes sense that OER production has no place outside of institutional funding. This is particularly true 
when there are few rewards for OER 

The main current features identified in ROER for promoting reuse are open licenses (specifically CC) 
and social networks. Intentionality, versioning and quality lag behind, and granularity and open formats 
are less evident. This suggests that repositories focus more specifically on OA licensing for OER and on 
taking care of or facilitating the creation of communities of users, who, in turn, could offer additional 
bottom-up quality criteria for deposited content. The reuse promotion pattern was slightly different in 
ROER created specifically to deposit OER. The open licensing indicator remained important, and the 
social network aspect increased considerably. In addition, quality and granularity began to play a more 
prominent role. Therefore, OER-exclusive repositories include a wider range of features, and use them 
more intensively, to facilitate the reuse of their resources. 

The most prevalent educational aspects offered in ROER are content type, format and subject. However, 

112 






















Repositories of Open Educational Resources: An Assessment of Reuse and Educational Aspects 
Santos-Hermosa, Ferran-Ferrer, and Abadal 


less than half of ROER provide specific metadata related to education, and just over one quarter include 
learning goals. In addition, the few ROER that provide educational metadata do not always use a specific 
educational standard such as LOM or SCORM, which were developed to better identify the pedagogical 
properties of OERs. However, some do incorporate LOM elements in other default metadata 
specifications, such as Dublin Core. This could be because many institutional repositories use DC as a 
general schema for describing their resources, since they were created to store not only educational 
material, but also research or other types of content. 

We could assume that educational features are more present in OER-exclusive repositories, which are 
created to meet an educational need. However, such repositories are not currently achieving their fullest 
potential. Although there is more educational information in this kind of repository (just over sixty per 
cent), there are still many cases of OER described and retrieved by type or format instead of by detailed 
educational metadata that better meets the users’ needs, for example the suggested use of content, its 
duration, pedagogy, target audience or learning goals. 

To sum up, current ROERs include more drivers that promote the reuse of OERs, mainly through open 
licenses and social networks, than features facilitating the retrieval and use of OERs according to 
educational needs, such as learning goals. 

A level of development of ROER was identified, based on the incidence of reuse and educational 
indicators. We identified four clusters of ROERs ranging from a small top level of repositories that 
include most of the reuse and educational dimensions to a large lower level of repositories that lack 
important features such as open formats, granularity and educational metadata. In the middle of this 
scale, we also identified a high category mainly characterized by providing social networks and 
educational metadata standards, and a medium category offering a diverse mix of indicators. 

We summarize here some of our findings in relation to previous studies. Some similar trends were 
identified in the general description of the ROER: most of them were multidisciplinary, institutional, 
predominantly based in Europe and the USA, and supported by DSpace technology. However, there are 
also a few fine distinctions, in three particular areas: in the numbers, insofar as our data show a 
significant increase in institutional ROER; in location, given that Europe and especially the UK, Spain 
and France took precedence over the USA in our study, while other related work found the opposite 
situation; and finally in technical infrastructure, where in line with other recent studies and in contrast 
to previous studies, our findings show an emerging tendency to adopt CMS. 

Some differences were also observed with respect to prior works. While our results showed a clear 
predominance of Dublin Core above other specifically educational metadata specifications, other 
studies found an opposite pattern or simply did not specify the metadata schema used by the ROER. 
Other differences observed in our research were increased openness in ROER and a more intensive use 
of CC licenses overall in areas with higher numbers of ROER, such as Europe and the USA. 

Finally, some of our main contributions were that OER-exclusive repositories led to better reuse and 
educational outcomes than hybrids and that most of the CC licenses used in ROER allowed reuse, as 
there was a notable presence of BY-NC-SA and BY-NC. This achievement, added to the use of other 
kinds of open license, shows increased openness of ROERs. 
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To further develop the research, the evaluation framework could be expanded by adding features, and 
the categorization of ROER could be examined in greater depth. The research could use indicators that 
are more focused on pedagogy and bottom-up approaches, as these increase the educational usefulness 
of ROER and the reuse of OER. Additional features could be based on users’ interaction, meaning 
communities and personal spaces, kind of information shared, social tagging procedures, etc., and types 
of OER, defined by whether the material was produced by students or educators and who it is aimed at 
(students, teacher trainers, the OER community, learning innovators). Users’ participation helps to 
evaluate and license the stored OER, which facilitates reuse and makes it easier to integrate educational 
needs into the classification and retrieval process. 
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